Enhanced analysis techniques using composite frequency spectrum data

ABSTRACT

Systems and methods are provided to evaluate moving objects undergoing periodic motion through the screening of a video recording of such objects in motion for frequency peaks in the spectral data, and to determine spatially where these frequencies occur in the scene depicted in the video recording, wherein a frequency spectrum is created for a subset of pixels or virtual pixels and a composite frequency spectrum table or graph is constructed of frequencies that are selected from among the larger group of frequencies represented by the frequency peaks of the spectral data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. patentapplication Ser. No. 17/356,617, titled “Enhanced analysis techniquesusing composite frequency spectrum data” filed Jun. 24, 2021, nowpatented as U.S. Pat. No. 11,282,213, and naming at least one of thepresent inventors, which claimed benefit of and priority to U.S. PatentApplication Ser. No. 63/043,299, which was filed on Jun. 24, 2020 andnamed at least one of the present inventors, the contents of all ofwhich are fully incorporated herein by reference.

BACKGROUND

The measurement of dynamic motion from civil structures, machines, andliving beings using video recordings from cameras has gained wideacceptance since 2010. The camera offers the advantages of being anon-contact sensor and provides information from millions of pixelssimultaneously. The light intensity measured at each pixel is a resultof the light reflected from the objects in the field of view in thevisible light range or radiation emitted from the objects due totemperature in the infrared portion of the electromagnetic spectrum.Other regions in the electromagnetic energy spectrum may providemeasurements related to other characteristics of the objects in thefield of view. In the visible light range changes of the light intensitycan be related to the motion of objects in the field of view. In somecases, a fundamental unit of measurement is displacement and theavailable accuracy achieved using video recordings is a tenth of a milor better. The application of mathematical techniques to magnify themotion and the ability to modify the frame rate on replay of therecorded video allows technicians to visually present the motion ofconcern and provides powerful arguments about what is happening and theneed for repair to decision-makers who may have limited technicalunderstanding of the underlying physics of the fault condition.

Prior to the use of video measurements of motion, the common practicewas to use sensors such as accelerometers or proximity probes to measurethe motion at each point of interest. Frequently, when monitoring amachine such as a motor-pump combination, a technician might measure themotion in all three axes, horizontal, vertical, and axial directions, ateach bearing housing. This process would yield a set of twelvemeasurements, on a four-bearing machine during a routine vibrationcheck. When troubleshooting, a more extensive set of readings might becollected at other positions on the machine train or its supportingstructure or foundation. This data is typically analyzed by reviewingthe time waveform data and the frequency spectrum at each of thepositions and axes at which data is collected.

This technique of individually comparing dynamic data at each locationof interest with other locations or historical data is tedious andrequires a good deal of expertise which is usually acquired over aperiod of years. This approach is manageable when the number of readingsis limited, for example less than 30 measurements. However, dynamicmeasurements using a camera often produces data from millions of pixels.This makes the method of manually comparing readings at each pixel oreven a set of composite pixels challenging or impractical. Although somepixels may not provide useful or meaningful information, there is theneed to identify a workable technique to accelerate the ability of theanalyst to efficiently screen the data from millions of pixels andquickly determine what are the significant frequencies of interest andwhere are they present spatially on the recorded images. Videomeasurement systems, such as the IRIS® MOTION AMPLIFICATION® System (RDITechnologies Inc. (Knoxville, Tenn.), allow the user to identify aregion of interest, ROI, using a graphical user interface. Both in thesesystems and in the present embodiments, an ROI can be considered as auser-selected portion of a field of view in a video recording. Withregard to prior video measurements systems such as those mentioned, thesystem software calculates the dominant motion in the ROI and presentstime waveform graphs with a cursor synchronized to the frames of thevideo as well as frequency spectra of the motion. This approach is quiteuseful when studying the behavior at known points of interest; however,a different method is desirable when screening a complex scene with manyelements in motion exhibiting multiple frequency components.

SUMMARY

There is a need to be able to efficiently screen a video for thefrequencies of the peaks in the spectral data and to determine spatiallywhere these frequencies occur in the scene captured by the camera. Theidentification of significant frequencies and their location in theimage can be facilitated by constructing a composite frequency spectrumgraph or table which combine the data from all pixels weighted in anynumber of ways to capture all significant frequencies in a single plotor table which is constructed from a matrix of frequency spectra or peaklists measured at each pixel in the recorded images. This provides theanalyst a single graph or table that shows all the significantfrequencies in the scene. A graphical user interface allows the analystto optionally select a frequency peak in this graph or table and thewindow containing the video interactively applies a color map thathighlights where the largest overall motion or the largest motion of theselected frequency appears in the scene. The colorized map applied tothe video can overlay a filtered video with enhanced visualization ofthe vibration such as, but not limited to, amplification of motion inthe recording. The colorization can use a single color or multi-colorscheme to indicate where the largest amplitude of this frequency occursspatially in the scene.

The composite frequency spectrum graph or table can be constructed usingany number of methods. The first step involves creating the timewaveform of the intensity changes at each pixel and then calculating theFFT of each waveform to produce a frequency spectrum that identifies thefrequency, amplitude, and phase values at uniformly spaced frequencypositions from zero frequency to a maximum frequency. The locations ofpeaks in the FFT frequency spectrum identify the frequencies with thelarger amplitudes for each pixel in the field of view. The entirefrequency spectrum of each pixel or the set of amplitude, frequency, andphase values at the larger peaks in the spectrum saved for each pixelprovide the basic data from which the composite spectrum graph or tableis constructed. A variety of techniques can be used to form thecomposite spectrum data, including linear averaging or peak holdaveraging, or other selective averaging techniques which compile a setof peak frequencies based on both amplitude and frequency of occurrencespatially in the video. The averaging method employed can be applied toeach line in the frequency spectrum of the individual pixels or only topeaks identified in the spectrum, or to only peaks with largeramplitudes or a having a higher occurrence count, i.e., a higherfrequency of occurrence. For example, if only frequencies that occur atleast 50 times are to be used in executing the steps on a particularvideo recording, this would be referred to as an occurrence threshold of50. The exact method for constructing the composite frequency spectrumgraph could be defined by a set of options specified by the user.Alternate embodiments may utilize a table of frequency peaks identifiedby the values of frequency, amplitude, and occurrence counts. Acomposite frequency spectrum graph could be constructed from this tableof peak values, or a composite spectrum table could be usedinteractively to apply color mapping to the video, which are among thenovel features described herein that are helpful toward viewing andassessing the motions of machines, parts, and components in complexarrangements. However, the objective will remain the same to present asingle composite spectrum graph or table which highlights thesignificant frequencies present in the scene and a graphical userinterface that facilitates visual identification in the field of view ofwhere these frequencies occur spatially. Herein, the terms “compositespectrum” and “composite frequency spectrum” are interchangeable andmean the same.

BRIEF DESCRIPTION OF FIGURES

The patent or application file with respect to the present disclosurecontains at least one drawing executed in color. Copies of this patentor patent application publication with color drawing(s) will be providedby the Office upon request and payment of the necessary fee.

The drawings, schematics, arrangements, figures, and descriptionscontained in this application are to be understood as illustrative ofsteps, structures, features, and aspects of the present embodiments.Accordingly, the scope of embodiments is not limited to features,dimensions, scales, and arrangements shown in the figures.

FIG. 1 is a composite frequency spectrum graph constructed by averagingeach line in the frequency spectra from all pixels in the scene capturedby the video.

FIG. 2 is a flowchart of how the data in a composite spectrum graph isconstructed.

FIG. 3A shows the region of interest selected by the user and marked bythe red rectangle on the still image of the rotor kit.

FIG. 3B illustrates the frequency spectra for the X and Y axes takenfrom the dominant motion detected in the region of interest marked bythe red rectangle on the still image of the rotor kit.

FIG. 4A shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for the testrotor kit with options set for a multi-color map.

FIG. 4B provides the interactive multi-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor the test rotor kit.

FIG. 5A shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for the testrotor kit with options set for a single-color map.

FIG. 5B shows the interactive single-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor the test rotor kit.

FIG. 6 presents the interactive options which may be specified by theuser to adapt the color map applied to the video.

FIG. 7A shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for this beltdriven machine with options set for a single-color map and specific tomotion at the selected frequency of 5 Hz.

FIG. 7B shows the interactive multi-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor the belt-driven machine occurring at the frequency of 5 Hz which isthe turning speed of the shaft.

FIG. 8A shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for this beltdriven machine with options set for a single-color map and specific tomotion at the selected frequency of 2.67 Hz.

FIG. 8B shows the interactive multi-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor the belt-driven machine occurring at a known frequency which is theturning speed of the belt.

FIG. 9A shows the interactive multi-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor a complex arrangement of equipment.

FIG. 9B shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for thiscomplex arrangement of equipment and documents that there is very littleactivity above 15 Hz and many closely spaced low frequency peaks.

FIG. 9C shows an expanded view of the composite frequency spectrum graphand highlights the detail in peaks present below 15 Hz and separates theclosely spaced, low frequency peaks.

FIG. 9D shows an even further expanded view of the composite frequencyspectrum graph and highlights the detail in peaks present below 15 Hzand separates the closely spaced, low frequency peaks.

FIG. 10A is a composite frequency spectrum graph constructed using thepeak hold technique that captures the largest amplitude value at eachfrequency in the spectrum from all the pixels in the field of view.

FIG. 10B is a composite frequency spectrum graph constructed from thesame video using the linear averaging technique at each frequency in thespectrum from all the pixels in the field of view.

FIG. 11 is a flowchart of how the data in a composite spectrum table isconstructed.

FIG. 12A is a table of the largest 20 peaks compiled from all pixels inthe field of view that have an occurrence count greater than 100 pixelsarranged in ascending frequency order.

FIG. 12B is a table of the largest 20 peaks compiled from all pixels inthe field of view that have an occurrence count greater than 100 pixelsarranged in descending amplitude order.

FIG. 12C is a table of the largest 20 peaks compiled from all pixels inthe field of view that have an occurrence count greater than 100 pixelsarranged in descending order of pixel count.

FIG. 13 is a flowchart of how the data in a composite spectrum table ispresented in a composite spectrum graph.

FIG. 14 is a composite frequency spectrum graph formed using the datafrom the table of the 20 largest peaks values from all the pixels in thefield of view.

FIG. 15A illustrates a composite frequency spectrum graph and FIG. 15Ban occurrence histogram, respectively, constructed from the compositespectrum table in FIGS. 12A-C.

MULTIPLE EMBODIMENTS AND ALTERNATIVES

In some embodiments within the scope of subject matter claimed herein, asystem is provided for evaluating a moving object undergoing periodicmotion. When sampled data is acquired, such as with a video acquisitiondevice, the data may exist in a video recording having a plurality ofvideo images of the moving object which are divisible into individualvideo image frames, and with each frame being divisible into a pluralityof pixels. Such a system or method may comprise or utilize a processorand a memory for storage of the individual video image frames as well asany that are modified through the processes described herein, and acomputer program operating in the processor, as well as one or morevideo acquisition devices. Embodiments are not limited to a particulartype of video acquisition device, but may include one or more videocameras, webcams, or digital cameras integral in cells phones. A videoacquisition device in the embodiments herein may be configured with anadjustable frame rate that allows the video images to be acquired at asampling rate that is sufficient to capture a plurality of frequenciespresent in the periodic motion. That is, video images are acquired by avideo acquisition device at a rate expressed in frames per second (fps),wherein for example at 120 fps there would be 1200 frames acquired in 10sec. A computer program in the embodiments herein comprisescomputer-readable program instructions executed by the processor and maybe configured to operate on a subset of pixels from the plurality ofpixels in a field of view of the video recording. In some embodiments,the computer program operates to create a frequency spectrum for eachpixel in the subset of pixels and to construct a composite frequencyspectrum, or a table, presenting one or more prominent frequencies(e.g., based on a higher amplitude) from among the plurality offrequencies in the field of view. A selection of a specific frequency inthe composite frequency spectrum graph or a table by the user willresult in a spatially identifying where this motion is present spatiallyin the field of view. There is some flexibility in steps taken toconstruct the composite spectrum graph or table which will be describedin the various embodiments presented herein. Other variations whichresult in a composite spectrum graph or table which is used to find thespatial location of specific frequencies would fall within the scope ofthis invention.

The construction of a composite frequency spectrum graph or table isbased on identifying the frequencies with larger amplitudes resultingfrom motion at all the pixels in the recorded video. This informationfrom millions of spatial locations is then intelligently reduced to asmaller set of values which highlight the significant frequencies ofmotion in the recording. The complete frequency spectrum or the set oflarger peaks in the spectrum at each pixel is retained so that colormaps can be applied to the video images to identify the spatial locationof large amplitudes of overall motion or the location of specificfrequencies of motion. The identification of motion at each pixel can bedetermined by constructing the waveform of the intensity changes at eachpixel location and then calculating the frequency spectrum at eachpixel, commonly accomplished using an FFT algorithm. The frequencyspectrum provides the frequency, amplitude, and phase values atuniformly spaced frequency positions from zero frequency to a maximumfrequency. The locations of peaks in the FFT frequency spectrum identifythe frequencies with the larger amplitudes for each pixel in the fieldof view. The overall magnitude of the motion at each pixel can becalculated from the time waveform data or the FFT spectrum. In analternate embodiment, the composite frequency spectrum could also beconstructed from a grid of pixels (an area of contiguous pixels in thefield of view as part of a defined section which may be rectangular)forming a less dense array of virtual pixels, or by tracking the motionof features in each section of the grid overlaying the field of view.This approach would produce waveform and spectrum data of the detectedmotion based for each section of the grid in both the x and y axes, asillustrated for the rectangular ROI in FIGS. 3A and 3B. The methods forconstruction of the composite frequency spectrum graph or table wouldremain the same, either all the spectra from both the x and y axes couldbe combined into a single composite frequency graph or table oralternatively a composite spectrum and table could be constructed foreach axis. By way of non-limiting example, the locating of features, forexample at the perimeter of a moving object, such as an edge, allows thesystem to perceive the effect of motion of the entire object. This couldbe a rocking motion of a motor, in which one side of the motor isdisplaced and a selected number of pixels working together as part ofthat feature are correlated in a manner to provide one way to perceivethe object in motion.

The movement of such a feature, and thereby that position on the movingobject, is associated with changes in location of the feature within thegrid, for example moving through a rectangular section of the grid orfrom one rectangular section into another, as determined by techniqueswhich are known in the art, such as optical flow algorithms that arewell known to those skilled in the art. In one aspect, each section ofthe grid as a unique location of the field of view can be thought of asa virtual pixel. In some embodiments related to this aspect, thecomputer program operates on at least one feature of the moving objectin the field of view by tracking movement of the at least one feature.The computer program then is able to produce a motion waveform and tocalculate a frequency spectrum for the at least one feature, and itconstructs a composite frequency spectrum presenting one or moreselected frequencies from among the plurality of frequencies, in which aselection of frequencies is based on predetermined criteria, such as aredescribed below in constructing a composite frequency spectrum graph ortable. In other aspects, all the pixels in a section of the gridcollectively are considered a virtual pixel, for example by taking anaverage of the frequencies and amplitudes determined for pixels within agrid section.

This set of frequency spectra from all spatial locations must becombined into a single graph or table which is presented to the user asa tool that he can use to select the frequencies of most interest to hisinvestigation. The user may select either the overall amplitude and oneof the significant frequencies in the field of view, and the system willuse supporting spectral data particular to each spatial locationdetermining how a color map is applied spatially to the video framesbased on user specified options. For high resolution video images, thisrepresents millions of individual frequency spectra. The amount ofsupporting information may be reduced by retaining only overallamplitude of motion and the amplitude, frequency, and phase values ofthe significant frequency peaks from each spectrum or by allowing theuser to selectively limit the field of view for analysis. The currentmethod for identifying spatially the location of large amplitude motionis to visually observe an amplified video and sequentially place ROIs atspatial locations where large amplitudes are observed or at locations ofknown interest such as a bearing housing. This tedious process will notshow where the same frequency of motion is present spatially everywherein the field of view. Also, the user may easily overlook possiblefrequencies of interest because he does not happen to put an ROI in oneof the spatial locations where the frequency occurs. The new methoddescribed herein brings all significant frequencies into a single graphor table and highlights the spatial locations which exhibit thisfrequency in a novel manner that changes the efficiency andeffectiveness of the investigative workflow followed by an analyst.

This method for identifying significant frequencies and their locationin the image depends upon the construction of a composite spectralfrequency graph or table which combines the data from all pixelsweighted in any number of ways that capture all significant frequenciesin a single plot or table. As described the graph or table is supportedby a set of complete frequency spectra or list of peak values from eachpixel or virtual pixel in the recorded images. Thus, when a specificfrequency peak is selected from the graph or table, the software canquickly search the supporting data stored for each pixel to determine ifthis frequency is present at this pixel and if so, convert the amplitudeto the appropriate color or to a single color to be assigned to thispixel or spatial location. The phase information could offer the abilityto adapt the color map applied to the video by showing only spatiallocations which share a specific phase relationship, such as in phase orout of phase at a selected frequency.

The construction of the composite frequency spectrum graph or tablerepresents an intelligent reduction of the information from potentiallymillions of spatial locations into a set of the most important frequencypeaks of interest to the analyst. The simplest approach is to linearlyaverage the amplitude values of the corresponding frequency line for thefrequency spectra from all pixels in the entire field of view or aselected subset of the scene. This can be effective if there are only afew significant frequencies present and there is adequate separationbetween frequencies when multiple frequencies are present. FIG. 1 is acomposite frequency spectrum graph constructed by averaging each line inthe frequency spectra from all pixels in the scene captured by thevideo. This graph shows a single strong peak around 15 Hz. Thisamplitude of this peak is so large that it causes all the other peaks toappear very insignificant when presented on a linear scale. In order toeffectively evaluate the smaller peaks, the data would need to bepresented on a log scale or the user would need to zoom in around thelower peaks by expanding the Y-axis scale. However, as the number offrequencies present in the field of view increases or there isinsufficient spacing between peaks in different pixels, then linearaveraging at each frequency line in the spectrum may cause peaks to maskeach other. Also, when using linear averaging on potentially millions ofpixels, smaller peaks may be lost due to background noise present atthese frequencies present in other pixels. In some embodiments, peakhold averaging may be preferred over a linear average. This techniqueretains the largest magnitude value which occurs at each line in thespectra from the individual pixels. FIG. 2 is a flowchart of how thedata in a composite spectrum graph is constructed. At box (or step) 21,within a field of view (or user-selected portion of the field of view),the system loop through pixels, which can also be present as virtualpixels, provided by a camera in the x- and y-directions. An intensitywaveform is produced for each pixel at box 22, and a FFT spectrum iscomputed for each at box 23. At box 24, a second loop through the pixelsor virtual pixels occurs in which, for each line in the frequencyspectrum, a new amplitude value is determined, for example by linearaveraging, or by selecting the largest value from among the pixels,through a combination of averaging and maximum peak value, or othermethods known in the art. At box 25, a composite frequency spectrumgraph can be constructed, providing a characterization of all dynamicmotion present in the field of view or selected portion, in a format offrequency on x-axis and amplitude values on y-axis, and therebyhighlighting visually one or more significant peak values in the sampleddata.

FIG. 3A shows the ROI, item 31, selected by the user and marked by thered rectangle on the still image of the rotor kit. An exemplary videoprocessing system which provides, among other features and capabilities,a user with an ability to draw a perimeter or region of interest withinthe video frame so that analysis may be focused on that region, isdescribed in U.S. Pat. No. 10,459,615 titled “Apparatus and method foranalyzing periodic motions in machinery” (Hay, Jeffrey R.; Oct. 29,2019), the contents of which are fully incorporated by reference hereinfor all purposes. FIG. 3B illustrates the frequency spectra for the Xand Y axes as measured in the directions orthogonal to the cameradetermined from tracking the motion of the dominant object detected inthe ROI marked by the red rectangle on the still image of the rotor kit.This is one of the methods for investigating the frequency of the motionpresent in the video. A second or third ROI selected by the user wouldpresent the frequency measured at those locations. If detailedanalytical information is desired at only a few locations, then thisapproach is sufficient for analysis. However, the analyst may missimportant information because he fails to look at the region of thescene where it occurs. The application of a color map could be generatedby identifying the frequency present in any selected ROI and requestingthe system to calculate the spectra at all other spatial locations toidentify where this frequency occurs in the scene; however the existenceof frequencies not present in the ROIs selected by the analyst couldeasily be overlooked. This approach may be viewed as an alternate, butpotentially less effective, embodiment that can highlight the spatiallocation of an identified frequency which still relies upon constructingfrequency spectra for all spatial locations.

FIG. 4A shows the composite frequency spectrum graph constructed fromthe intensity changes at every pixel in the recorded video for the testrotor kit with options set for a multi-color map. Not all of thefrequency peaks in the frequency spectra in FIG. 3B appear in thecomposite frequency spectrum graph which has been constructed by using alinear average of all pixels. FIG. 4B displays the interactivemulti-color map which highlights the largest magnitude intensity changesat every pixel in the recorded video for the test rotor kit. This colormapping is applied to the overall motion rather than to any specificfrequency because the user has not selected the option to “Displayfrequency specific motion contours,” labelled as item 41. FIG. 5A showsthe composite frequency spectrum graph constructed from the intensitychanges at every pixel in the recorded video for the test rotor kit withoptions set for applying a single-color map. FIG. 5B displays theinteractive single-color map which highlights the largest magnitudeintensity changes at every pixel in the recorded video for the testrotor kit.

The application of a specific color map to the video frames iscontrolled by user-specified options as shown in FIG. 6 . The user canspecify a single color or multi-color map, items 67 and 68,respectively. Sliders on the color bar, item 62, to the right of thegraph determine how the amplitude of the motion is scaled to a singlecolor or multiple colors. An Autoscale option, item 61, willautomatically scale the colors based on the population distribution ofthe amplitude range in the overall motion or the motion at the frequencyselected for presentation. There is a popup option, item 65, thatappears as a result of a right mouse click. The popup options allow theuser to specify whether pixels with amplitude values above or belowthose represented at the upper and lower sliders, items 63 and 64respectively, will be assigned a color or left black and white in thereconstructed video. If the user selects the Single Color map option,item 67, then a red color is applied at each pixel based on theamplitude range represented by the slider positions and the magnitude ofthe overall motion or the magnitude of a selected frequency for thatpixel. The pixel is left with the original grayscale value if it doesnot meet the specified amplitude value. In this case, the red color isapplied at all spatial locations with higher amplitudes of motion ifFill Above is specified and the red color is applied over a selectedrange of amplitudes if Fill Above is not selected. In a multi-color map,the color spectrum is mapped to a range of amplitude values with blueapplied at lower amplitudes and red applied at larger magnitudes at eachpixel or spatial location. In the multi-color case, when the Fill Aboveoption is specified, the pixels with higher amplitude motion asdetermined by the upper slider will be given a red color. When the FillBelow is selected, then the pixels with lower amplitude values asdetermined by the lower slider will be assigned a blue color. Pixelswith an amplitude of motion between that specified by the sliders willbe assigned an appropriate color from the color spectrum. Thetransparency bar, item 66, when active will allow the user to set avalue between 0 and 100. A transparency value of 0 will not show any ofthe grayscale images in the video and the entire scene will be filledwith color based on the user selections. A transparency value of 100will show only the original grayscale values of the pixels in the video.Intermediate values create a mix of the grayscale and color values.

FIG. 7A shows the composite frequency spectrum graph constructed bylinearly averaging the intensity changes at every pixel in the recordedvideo for this belt driven machine with options set for a single-colormap and specific to motion, item 71, at the selected frequency of 5 Hz,item 72. FIG. 7B shows the interactive single-color map which highlightsthe largest magnitude intensity changes at every pixel in the recordedvideo for the belt-driven machine occurring at the frequency of 5 Hzwhich is the turning speed of the shaft. The color map clearly indicatesthe shaft that is moving at this frequency. FIG. 8A shows the compositefrequency spectrum graph constructed by linearly averaging the intensitychanges at every pixel in the recorded video for this belt drivenmachine with options set for a single-color map and specific to motionat the selected frequency of 2.67 Hz. FIG. 8B shows the interactivesingle-color map which highlights the largest magnitude intensitychanges at every pixel in the recorded video for the belt-driven machineoccurring at the frequency of 2.67 Hz which is the turning speed of thebelt. The color map clearly indicates that the belt is moving at the2.67 Hz frequency.

FIG. 9A shows the interactive multi-color map which highlights thelargest magnitude intensity changes at every pixel in the recorded videofor a complex arrangement of equipment. FIG. 9B shows the compositefrequency spectrum graph constructed by linearly averaging the intensitychanges at every pixel in the recorded video for this complexarrangement of equipment and documents that there is very littleactivity above 15 Hz and many closely spaced low frequency peaks. FIG.9C shows an expanded view of the composite frequency spectrum graph andhighlights the detail in peaks present below 15 Hz and separates theclosely spaced low frequency peaks. FIG. 9D shows an even furtherexpanded view of the composite frequency spectrum graph and highlightsthe detail in peaks present below 15 Hz and separates the closelyspaced, low frequency peaks.

The ability to use the composite frequency spectrum graph andinteractively study the gross motion or to isolate the motion at asingle frequency and visually locate the position where this occurs inthe scene is a novel technique that enables the analyst to quicklyreview a video recording and focus his attention on the importantspatial locations in the scene and identify all of the components movingat the same frequency. The ability to view single or multi-color colormaps is typically a user preference and either map presents the sameinformation. The threshold slider, item 42, shown in FIG. 4A and otherfigures, allows the user to select what amplitude of motion isrepresented in the colorization and provides instantaneous changes asthe user moves the slider. The transparency option specifies how much ofthe image is visible behind the color map. The color map shown in thefigures presented is superimposed over a single frame of the video.However, the color map will also remain during the video replay with orwithout amplification. This allows the user to visualize the motion inthe presence of the color map to focus attention to the correct spatiallocation for the frequency of interest. In some embodiments, softwareproviding machine-readable program instructions may be used to applytechniques such as MOTION AMPLIFICATION® for a better visualization ofthe rotating component. An exemplary system providing such capabilitiesis described in U.S. Pat. No. 10,062,411 titled “Apparatus and Methodfor Visualizing Periodic Motions in Mechanical Components” (Hay, JeffreyR. et al.; Aug. 28, 2018), the contents of which are fully incorporatedby reference herein for all purposes. Among other features andcapabilities, this patent describes multiple embodiments in which a newimage sequence or sequences are computed, showing the movements of anobject(s) in motion being visually amplified, the contents of which areincorporated herein by reference in their entirety.

The video can also be filtered to a specific frequency such that onlythe motion at that frequency is present in the video. The ability toamplify the motion which has been filtered to a single frequencyprovides the analyst a unique ability to visualize a specific motion ofconcern and present this to interested stakeholders.

Other methods of constructing a composite frequency spectrum graph ortable may be applied in alternate embodiments. The purpose of thecomposite frequency spectrum graph is to highlight the most significantfrequencies present in the scene recorded by the video by increasing theprominence of motions associated with more significant frequenciescompared to frequencies associated with more routine motions, such asbaseline vibrations or noise. In this regard, in some embodimentscriteria are predetermined and applied by the system to identify themost significant frequencies among those represented by frequency peaksin the composite frequency spectrum. Non-limiting examples of suchcriteria include those frequency peaks with the largest amplitudes orthose which occur in a large number of spatial locations in the field ofview being analyzed. Alternative methods described in this applicationrepresent other useful techniques for constructing the compositefrequency spectrum graph or table. In addition to linearly averaging thespectra from all pixels, a peak hold summation can be used to improvethe visualization of scenes with a complex set of frequencies present.Additionally, an alternate method that maintains excellent frequencyresolution and identifies spatially meaningful frequencies with anoccurrence counter can be accomplished by reducing the complete spectrumat every pixel location to a set of the largest N peaks defined by aprecise frequency, amplitude and phase; and then create the compositefrequency spectrum data by scanning the peak data stored for each pixeland retaining the largest M peaks which occur at a minimum number ofpixels. The various embodiments for constructing the data needed tocreate a composite frequency spectrum or table described herein areillustrative and not intended to be limiting. Any number of variationscould be devised by one skilled in the art to construct the data whichcharacterize the set of significant frequencies in the video recordingwhich accomplish the intended purpose of providing the analyst astraightforward method for identifying these frequencies spatially inthe recorded scene and would fall within the scope of this invention.

A composite frequency spectrum graph from a video recording may becomposed from 1 to 12 million pixels or more depending on the cameraused to acquire the video. A straight linear average of several millionvalues can easily obscure peaks which occur in a small subset of thescene. It has been observed in some data sets that this approachfrequently results in a large value at the lowest frequency in thespectra with a descending slope as the frequency increases as seen inFIG. 9B. On other occasions, a single peak is dominant and most of theother peaks are barely visible above the floor noise in the spectrum asseen in FIG. 1 . High floor noise in the frequency spectra of pixelswhich do not contain the frequency of interest can contribute to maskingpeaks in composite frequency spectrum graph. A linear scale is notoptimum in this situation. The presence of the smaller peaks would bemore apparent using a log amplitude scale. Alternatively, the user mayelect to use the zoom features of the graphical interface to select thesmaller peaks which are barely visible over the floor of the compositefrequency spectrum graph as illustrated in FIGS. 9C and 9D.

Other methods may be better suited to retain the largest frequency peaksin the data or to retain the large peaks which occur at a specifiedminimum number of pixels. One alternative is to use a peak hold methodwhere the maximum amplitude value at each frequency location in thespectrum is retained rather than constructing the mean value. This willensure that the largest frequency peaks in the scene are visible forreview regardless of how many pixels share the largest peak frequencies.However, pixels whose spectra are mostly composed of noise rather thanmeaningful frequencies may still mask smaller peaks in the correspondingpixels. FIG. 10A is a composite frequency spectrum graph constructedusing the peak hold technique that captures the largest amplitude valueat each frequency in the spectrum from all the pixels in the scene. Thiscan be compared FIG. 10B which is a composite frequency spectrum graphconstructed from the same video using the linear averaging technique ateach frequency in the spectrum from all the pixels in the scene.Clearly, the composite frequency spectrum graph constructed with peakhold averaging shows frequency peaks that are not present in thecomposite frequency spectrum graph formed from linear averaging. Onesuch peak has been labeled 81 in FIG. 10A.

Complex scenes such as shown in FIG. 9A often contain many frequencieswhich are closely spaced. The approach of combining amplitudes from eachline in the spectrum by averaging or using some other technique such aspeak hold averaging can obscure peaks because a peak in the spectrum maytypically involves 3-5 adjacent frequency values. Closely spacedfrequency peaks could partially overlay each other. An improved methodwould focus on the significant peaks in the spectrum only and ignore thedata from frequency locations that do not contain peaks or only peakswith insignificant amplitude. These areas typically represent noise anddo not contribute significant amplitude to the signal from that pixel.Typically locating the top 15-20 peaks in a spectrum will capture 99% ofthe total amplitude present in the spectrum. Additionally, a largefrequency peak which occurs at only a few pixels is probably notmeaningful in the motion in the scene. It is unlikely that a realvibration would only appear spatially in a few pixels, for example 100pixels or less. Thus, it is important to determine how many pixelsexhibit a specific frequency.

Another alternative embodiment would be to locate the largest Nfrequency peaks for each pixel in the scene. These peak values from eachpixel may be combined by applying both amplitude and frequency ofoccurrence criteria to determine which frequency peaks are retained toreconstruct a single composite frequency spectrum graph or to create anordered composite frequency spectrum table that could be used toestablish the color mapping applied to a single frame or a replay of thevideo. FIG. 11 is a flowchart of how the data in a composite spectrumtable is constructed. The overall amplitude value of each pixel isretained in order to produce a magnitude color map that is not frequencyspecific.

The lowest value in the spectral data or the average of floor valuesselected from non-peak locations in the frequency spectra would bechosen as the floor value assign to any frequency line where no peakexists in the data. Peak values would be superimposed over the floorvalues as a single line or possibly as two or three lines. Peaks,defined by a frequency, amplitude, and phase value, could be determinedin any number of logical ways. One approach would be to locate thelargest N peaks in the FFT spectrum of each pixel, box 94 in FIG. 11 .Prior to doing this, the steps as shown in boxes 91, 92, and 93 of FIG.11 would be the same as those for steps 21, 22, and 23 discussed withFIG. 2 . The frequency and amplitude of the located peak could bedefined by the frequency value at the largest amplitude, by applyingfitting techniques to the top values forming the peak, or by calculatingmore accurate values based on the window function applied to the timedata before the FFT is calculated, using conventional methods known tothose skilled in the art. As desired, at box 95 the N largest peakvalues and lowest floor value for each pixel are stored in memory. Asecond pass screening of the N largest peaks for each pixel would locatethe largest M peaks in the entire set of the pixels, box 96 in FIG. 11 ,and a third pass determine the number of pixels exhibiting the samefrequency peak for this set of M largest peaks, box 97. The amplitude ofan individual peak could be established by calculating the mean of theamplitude values for all the pixels where this frequency occurs or usingthe maximum value in this set, or some combination of the mean and themaximum value. This creates a table of M rows, with each row containinga frequency value, an amplitude value, and a total pixel count. As theexecution of steps according to a flowchart represented by FIG. 2presented a constructed composite frequency spectrum graph, in FIG. 11at box 98 a table of significant peaks is presented providing acharacterization of all dynamic motion present in the field of view orselected portion, the results of which can be presented in order offrequency, amplitude, or occurrence.

FIG. 12A is a table of the largest 20 peaks compiled from all pixels inthe field of view of the complex arrangement shown in FIG. 9A that havean occurrence count greater than 100 pixels arranged in ascendingfrequency order. Some additional columns are present to show the rank ofthe peaks by frequency or amplitude, and to present frequency values inorders as well as Hz. FIG. 12B is a table of the largest 20 peakscompiled from all pixels in the field of view that have an occurrencecount greater than 100 pixels arranged in descending amplitude order.FIG. 12C is a table of the largest 20 peaks compiled from all pixels inthe field of view that have an occurrence count greater than 100 pixelsarranged in descending order of pixel count. All of these forms of thetable are useful to the analyst and facilitate his evaluation of whatfrequencies he should emphasize and in what order he may want to reviewthe data. Through conventional techniques, the table can be rearrangedto allow a user to change the presentation by double clicking the columnheading to sort by the data in that column. This table is an alternaterepresentation of the same information in the composite frequencyspectrum graph and can be used without the graphical composite frequencyspectrum graph by double clicking on any row as a method for selectingthe frequency to colorize in the video. FIG. 14 is a composite frequencyspectrum graph constructed formed using the data from the table of the20 largest peaks values from all the pixels in the scene. FIG. 13 is aflowchart of how the data in a composite spectrum table is presented ina composite spectrum graph. It should be noted that each frequency peaklocated in the spectra of the individual pixels also has a phase valueassociated with it. These phase values could be used to create colorizedphase maps for a selected frequency. The phase map at a selectedfrequency would overlay a selected frame of the video and allow the userto understand phase relationships between objects in the field of view.Phase maps would color pixels with a specified phase relationship or usea color spectrum to identify phase variation from in-phase toout-of-phase. A U.S. patent application which provides exemplarydescriptions of a system for evaluating a moving object undergoingperiodic motions is Ser. No. 17/217,299, filed Mar. 30, 2021, titled“Apparatus and Method for Visualizing Periodic Motions in MechanicalComponents,” the contents of which are fully incorporated by referenceherein for all purposes. Among other features and capabilities, thisapplication describes a system that calculates and displays phase valuesfor pixels at a selected frequency.

The composite spectrum tables in FIGS. 12A-C contain the occurrencecount in pixels, which is not present in the composite frequencyspectrum graph. This information could be added to the compositefrequency spectrum graph as a histogram graph shown below the frequencyaxis of the composite frequency spectrum graph as described in step 104in FIG. 13 . For example, in box 101 of FIG. 13 an amplitude value forall lines in the frequency spectra can be stored to the minimum floorvalue located in the search of all pixels. Then at box 102, for each ofthe M located peaks in the composite frequency spectrum table, a peakamplitude is assigned to the frequency line closest to the peakfrequency. Alternatively, at box 102, the system assigns a calculatedamplitude value to some number of the frequency lines (e.g., 2 or 3)closest to the peak frequency value based on the FFT window formulas tolocate peaks which do not fall precisely on a spectral line. Thecomposite spectrum data characterizing all dynamic motion present in thefield of view or selected portion can be graphed (box 103) in a formatof frequency on the x-axis and amplitude on the y-axis to visuallyhighlight significant peak values in the sampled data. An occurrencehistogram (box 104), then is one potential tool for characterizingfrequencies present in a field of view or portion thereof.

In this regard, FIG. 15B adds an occurrence histogram as complementaryinformation to the composite frequency spectrum graph shown in FIG. 15A,which is a copy of the one shown in FIG. 14. As desired, the compositefrequency spectrum graph (FIG. 15A) and occurrence histogram (FIG. 15B)may be displayed together. The data presented in the table or thecomposite frequency spectrum graph could be customized by the user bycontrols which allow the user to specify the number of peaks retainedand the minimum occurrence count. The discussion of this processingtechnique has indicated that all pixels in the recorded field of viewwould be processed; however this technique could be applied to a subsetof the pixels in the field of view as selected by the user using agraphical interface to identify pixels to be included or excluded. Theinteractive graphs or table of peak values combined with colorization ofa single frame or the entire video on replay provide a powerful tool forinvestigating the dynamic information available from millions of pixelsenabling the analyst to quickly locate and visualize any motion ofinterest in the recorded video. Once a frequency value is selected fromthe graph or the table, then the frequency spectrum or saved peaksassociated with each pixel are searched to determine if the frequency ispresent at this pixel and, if present, its magnitude. This determinesthe colorization applied to each pixel.

At times, it may be beneficial to eliminate areas within the image forinclusion into the composite frequency spectrum graph. Various reasonsfor their exclusion could exist but an exemplary list may include butnot be limited to: areas known to have no motion, areas of no targetedsubject matter such as the floor, the sky or other background, areaswith obscuring phenomenon such as steam, or areas saturated by light.These areas may be graphically defined by the user as regions set toinclude pixels into the composite frequency spectrum graph and regionsset to excluded pixels from the composite frequency spectrum graph.These regions may be determined by shapes drawn on the screen to definethese regions. These shapes may be color coded or shaded to indicateinclusion or exclusion.

The composite frequency spectrum graph may be limited to specificobjects, object types or other limiting subject matter. Imagesegmentation may be used to narrow the region or pixels to be includedwithin the composite frequency spectrum graph or table. It will beappreciated that image segmentation is a well-known and establishedtechnique, which is familiar to persons of ordinary skill in this field.The user may employ image segmentation by selecting an area in the imageand the segmentation process includes all areas associated with thatsegment. For example, the user may click on a pipe and the imagesegmentation process identifies all areas in the image associated withor a part of that pipe for inclusion. Object recognition may be anotherway in which the areas in which pixels are included into the compositefrequency spectrum graph are limited or narrowed. It will be appreciatedthat object recognition is a well-known and established technique whichis known to persons of ordinary skill in this field. The user may wishto look at only the motor or pump or both. Object recognition may beemployed to identify the pixels associated with the motor. The objectrecognition may identify the object, and the pixels associated with theobject and only include that object. Object recognition may also be usedto identify an object type, for example, pipes so that all pipes in thescene are included.

The composite frequency spectrum graph or table may also be used to forma set of filtered data videos. From the composite frequency spectrumgraph or table, a list of frequencies can be derived, for example likethose presented in FIG. 12A-C. The user may select the X frequencies oflargest amplitude in the composite frequency spectrum graph, the highestfrom a peak hold or other set of criteria that are common in selectingpeaks of interest. These selected frequencies may be used to createindividual videos filtered at those frequencies with color mappingoptionally applied to facilitate rapid, prioritized review of thesignificant frequencies in the original video.

It will be understood that the embodiments described herein are notlimited in their application to the details of the teachings anddescriptions set forth, or as illustrated in the accompanying figures.Rather, it will be understood that the present embodiments andalternatives, as described and claimed herein, are capable of beingpracticed or carried out in various ways. Also, it is to be understoodthat words and phrases used herein are for the purpose of descriptionand should not be regarded as limiting. The use herein of such words andphrases as “including,” “such as,” “comprising,” “e.g.,” “containing,”or “having” and variations of those words is meant to encompass theitems listed thereafter, and equivalents of those, as well as additionalitems.

Accordingly, the foregoing descriptions of several embodiments andalternatives are meant to illustrate, rather than to serve as limits onthe scope of what has been disclosed herein. The descriptions herein arenot intended to be exhaustive, nor are they meant to limit theunderstanding of the embodiments to the precise forms disclosed. It willbe understood by those having ordinary skill in the art thatmodifications and variations of these embodiments are reasonablypossible in light of the above teachings and descriptions.

What is claimed is:
 1. A system for evaluating a moving objectundergoing periodic motion using at least one video acquisition devicethat acquires sampled data in a video recording, the video recordinghaving a plurality of video images of the moving object which aredivisible into individual video image frames, and with each frame beingdivisible into a plurality of pixels, comprising: a processor and amemory for storage of the individual video image frames; and a computerprogram operating in said processor, wherein the video acquisitiondevice is configured with an adjustable frame rate that allows the videoimages to be acquired at a sampling rate that is sufficient to capture aplurality of frequencies present in the periodic motion; and wherein thecomputer program operates on a subset of pixels from the plurality ofpixels in a field of view of the video recording to create a frequencyspectrum for each pixel in the subset of pixels, each frequency in thefrequency spectrum having a peak, to construct a composite frequencyspectrum presenting one or more selected frequencies from among theplurality of frequencies wherein the selection of frequencies is basedon predetermined criteria, and to store a phase value for each peakassociated with each pixel in the subset of pixels.